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Executive Summary 



I n fall 2003, the Head Start Bureau began implementing the Head Start National 
Reporting System (NRS), an ambitious initiative to assess systematically the early literacy, 
language, and numeracy skills of all 4- and 5-year-olds enrolled in Head Start. Required 
as part of the administration’s Good Start, Grow Smart initiative, the NRS aims to collect 
information on a standard set of child outcomes from all Head Start programs in a 
consistent manner. Head Start has a decade-long history of concentrating on child outcome 
measures within its performance expectations. Specific national program performance 
measures were developed in 1995 and, in 1996, the Head Start Program Performance 
Standards strengthened requirements for the ongoing screening and assessment of children 
throughout their Head Start participation. In 1998, after the Head Start reauthorization, all 
programs were required to include child outcomes in their self-assessment process by 2003. 
In addition to using child assessment to assess program performance, the Head Start Bureau 
supports large scale research projects that assess children’s performance and experiences 
within Head Start, using representative samples. These studies include the Head Start Impact 
Study (HSIS) and the Family and Child Experiences Survey (FACES). 

Analysis of the NRS data will enable the Head Start Bureau to determine how children 
progress on a Kmited set of outcomes during the year preceding kindergarten. The data will 
also provide the Bureau with information that can be used to enhance its current program 
monitoring system and develop targeted training and technical assistance. The NRS includes 
a 1 5-minute child assessment battery, a system for training staff from all Head Start grantees 
to administer the assessment, and a computer-based reporting system that programs use to 
enter the completion status of assessments and report information on the characteristics of 
participating Head Start programs, teachers, and children. The Head Start Bureau provides 
each program with a summary report of average results for all children in the program who 
were assessed, which is available after each fall and spring administration. Reference tables 
are also created to allow programs to compare their scores to national averages, to regional 
averages, and to programs similar to them based on a number of characteristics, such as the 
percentage of children who are Enghsh Language Learners or the program auspice. 

In July 2003, the Administration on Children, Youth and Famikes (ACYF) contracted 
with Mathematica Policy Research, Inc., (MPR) and its subcontractor Juarez & Associates 
(J&A) to conduct the Head Start NRS Quality Assurance and System Development Project. 
The project had two components — an implementation study to assess the quahty and other 
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aspects of the first year of NRS implementation (training, child assessment, data entry, and 
program perspectives) and support for system development activities that could enhance the 
quahty and usefulness of the NRS. A final report on the Year One QuaKty Assurance Study 
was submitted to the Head Start Bureau in December 2004 (Paulsell et al. 2004).' This report 
documents Head Start programs’ experiences with the NRS during the spring of its second 
year of implementation. The report is based on information collected through visits to a 
nationally representative sample of 35 Head Start programs in spring 2005. During these 
visits, MPR and J&A observed a sample of child assessments and interviewed staff about 
their experiences implementing the NRS, particularly in spring 2005. 



Design and Development of the Head Start National Reporting System 

Head Start has long emphasized continuous program improvement and outcomes- 
oriented accountability. With an extensive history of conducting research and program 
evaluations. Head Start began developing specific program performance measures in 1995, 
in part to be responsive to the Government Performance and Results Act (GPRA) 
requirements. In 1996, the Family and Child Experiences Survey (FACES) was launched to 
collect data on the performance indicators. The 1996 revisions of the Head Start Program 
Performance Standards instituted new requirements for the screening and assessment of 
children for the purpose of improving teaching and learning. 

Following the 1998 reauthohzation of Head Start, the Head Start Bureau required all 
programs to include child outcomes in their self-assessment process by 2003. In August 
2000, the Bureau issued an information memorandum that laid out the steps programs must 
take to meet this requirement. These results-based standards and performance measures 
were presented in the context of a “Head Start Child Outcomes Framework” comprising 
eight general developmental domains, including several — for instance, language 
development, literacy, and mathematics — that overlap with the assessment requirements of 
the NRS (Administration on Children, Youth and FamiKes 2003). The child assessments 
required of all programs encompass 1 3 legislatively mandated indicators in language, Kteracy, 
and mathematics. In implementing these assessments over the past four years, all Head Start 
grantees have been charged with (1) improving the objectivity of their assessments, 
(2) analyzing the data over time in order to understand the nature and patterns of children’s 
progress, and (3) incorporating the results into continuous program improvement efforts. 
To meet these requirements. Head Start programs were permitted to select their own 
assessment instruments, as long as their instmments measured progress in the required 
developmental domains. Programs currently use a wide range of assessment strategies and 
tools to measure children’s progress. 



* In July 2005, oversight of the contracts for the Head Start NRS QuaKty Assurance and System 
Development Project was transferred from the Head Start Bureau to the Office of Planning, Research, and 
Evaluation (OPRE), Administration for Children and FamiKes, U.S. Department of Health and Human 
Services. 
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In addition to conducting these locally-selected assessments of children’s outcomes, 
programs have also heen charged with implementing the NRS, the first national assessment 
system for all 4- and 5-year-olds enrolled in Head Start. Initiated in April 2002 when the 
Bush administration announced the Good Start, Grow Smart early childhood initiative, the 
NRS is a key element of the “Strengthening Head Start” component of this initiative. It 
creates a national assessment and reporting system out of the congressionally mandated 
“standards of learning,” thus carrying out the President’s directive to develop “a strategy to 
ensure that, for the first time, every Head Start center assesses the standards of learning in 
early literacy, language, and numeracy skills.” As the Assistant Secretary for Children and 
Famihes noted, “The President’s Good Start, Grow Smart initiative challenges us to improve 
the operational effectiveness of Head Start programs hy developing a systematic, nationwide 
approach to assessing every child’s school readiness” (Horn 2003). 

To aid in developing the NRS, in August 2002 the Head Start Bureau contracted with 
Westat, Inc., and its subcontractor Xtria, LLC. The contractors convened a Technical Work 
Group (TWG) of 16 experts in child development, child assessment, measurement, and 
program evaluation. Beginning in December 2002, the TWG met three times to advise the 
contractors on the design of the NRS and the selection of instmments to be included in the 
child assessment. In addition, the Bureau convened several discussion sessions, focus 
groups, and workshops with Head Start program staff, early childhood researchers, and 
assessment experts to discuss plans for the NRS. 

The NRS Assessment 

In April and May 2003, Westat field-tested an initial child assessment battery with 
1,434 children in 36 programs. The field test assessed the measurement of several school- 
readiness domains, as defined by congressional and Presidential mandates. The assessment 
was designed with the goal of appraising skills that are critical to elementary school 
achievement, can be readily enhanced within the Head Start setting, are supported by parents 
of Head Start children, and are mastered by a majority of non-low-income children prior to 
kindergarten. Based on the analysis of data from the field test, and considering input from 
the TWG and others, ACYF approved a 15-minute NRS assessment battery for fall 2003 
that contained four components: 

1 . Comprehension of Spoken English (Language Screener). This component 
serves as an EngKsh-language screener to identify children whose EngKsh is 
insufficient to participate in the full assessment. It is composed of two subsets 
from the Oral Language Development Scale (OLDS) of the Preschool 
Language Assessment Scale (PreLAS) 2000 (Duncan and DeAvila 1998). The 
first set of items uses the “Simon Says” game to request that children follow 
simple commands, such as “touch your ear” and “point to the door.” In the 
second set of items, children are asked to name or describe the function of 
objects in pictures. 
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2. Vocabulary. Adapted from the third edition of the Peahody Picture 
Vocabulary Test (PPVT-III; Dunn and Dunn 1997), this section includes 
24 items that represent a range of difficulty. 

3. Letter Naming. A test developed hy Westat for the Head Start Quality 

Research Centers Consortium (ZiU 2003h), this section presents all 26 pairs of 
upper- and lower-case letters of the alphabet in three groupings (with 30 letters 
in the Spanish version). Children are asked to identify the letters they know 
by name. 

4. Early Math Skills. Adapted from the mathematics assessment used in the 

Early Childhood Longitudinal Study — Kindergarten cohort (ECLS-K), this 
section includes items on number understanding; shape recognition; relative 
size judgments and measures; and simple word problems involving reading 
graphs, counting, or basic addition and subtraction (Zill 2003a) . 

A Spanish-language version of the child assessment was also developed. During the 
first year of NRS implementation, all children whose home language was identified as 
Spanish were assessed in both English and Spanish, provided they passed the language 
screener for each version of the assessment. All children took the English-language 
assessment first. Spanish-speaking children who did not attain the threshold on the English 
language screener were assessed in Spanish only. During the second year of implementation, 
children whose home language was identified as Spanish were still assessed in both 
languages, as long as they passed the language screener for each version. However, 
beginning in fall 2004 in response to feedback from the TWG, Head Start Bureau staff, and 
local Head Start staff who participated in the Year One Quality Assurance Study, children 
whose home language was identified as Spanish took the Spanish-language assessment first, 
followed by the English version. 

Since first implementing the NRS in fall 2003, Westat has made minor changes to the 
NRS battery based on recommendations from the TWG and findings from MPR’s reports 
on the Head Start NRS Year One Quahty Assurance Study. These changes have included 
shortening introductions, changing some words in the vocabulary section, simplifying the 
directions on the letter-naming section, improving some pictures in the math section, 
simplifying question wording in that section, and simplifying the hand gestures required for 
some items. In addition, Westat added clarification to training materials to allow some 
minor deviations in the script; these enable assessors to help children feel more comfortable 
during the assessment. 

The Computer-Based Reporting System 

The Head Start Bureau implemented the Computer-Based Reporting System (CBRS) to 
collect background information on Head Start programs and children and facilitate the 
identification of ekgible children and tracking of completed assessments. The CBRS is a 
web-based system into which Head Start program staff enter all relevant information. 
Included in the program-level data are contact information for the grantee, delegate agencies. 
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centers, and program start and end dates. Classroom-level information includes the type of 
class (such as part-day or full-day), total enrollment, and number of classroom staff 
Information on teacher qualifications and experience is also collected. For each ehgihle 
child, staff enter the date of hirth, classroom entry date, years in Head Start, disahihty status, 
language spoken at home, level of EngKsh proficiency, ethnicity, race, and assessment 
completion status. The CBRS is used to assign identification numbers and can print out 
class rosters for use in tracking assessments, as well as assessment completion reports. 

Beginning in the second year of NRS implementation, Xtria, the subcontractor 
responsible for developing the CBRS, expanded its reporting capabilities and system 
functions. Local programs can now generate assessment completion reports at the program 
or center level; perform data searches; view and operate the CBRS in Spanish; and take 
advantage of both a “data copy” feature, to reduce data entry duplication for certain fields 
that remain the same from the previous program year, and a data import feature, to import 
data from the program’s management information system. Centers can also enter or make 
changes to data for multiple children at one time. In addition, grantees can produce NRS 
data reports from the CBRS about their delegate agencies. 

NRS Quality Assurance and System Development Project 

As noted, the NRS is the first implementation of a nationwide assessment of 4- and 
5-year-olds enrolled in Head Start. ACF conducted the 2003 field test with 1 ,434 children in 
the spring of 2003 to provide a foundation for expanding nationwide, and funded the NRS 
Quahty Assurance and System Development Project as part of its commitment to making 
ongoing improvements. The Head Start Bureau contracted with MPR and J&A to assess the 
accuracy and fidehty with which locally trained Head Start program staff across the country 
administered the assessment protocol. In addition, the Head Start Bureau sought to learn 
more about local program perspectives on the assessment process. 

Results of the Year One Quahty Assurance Study demonstrated that, while there was 
room for improvement. Head Start staff were able to administer the NRS assessment with a 
fairly good degree of fidehty to the protocol (Paulsell et al. 2004). Moreover, analyses of site 
visit interview data and assessment observations yielded a number of helpful suggestions for 
improving the assessment battery and process. The Head Start Bureau had several goals for 
the second-year study. First, Bureau staff wanted to determine whether the level of fidehty 
documented through the Year One study would be sustained over time. Second, staff 
wanted to determine whether comparable levels of fidehty would be observed in different 
samples of programs (and consequently requested that MPR draw different samples in fall 
and spring of the second year, for a total of 70 programs). Third, the Bureau wanted to 
collect ongoing information about programs’ experiences and perspectives on NRS 
implementation, including their perspectives on changes made to the assessment process 
from Year One to Year Two. 

In addition to conducting the Quahty Assurance Study, MPR and J&A were charged 
with recommending system development strategies for enhancing the quahty and usefulness 
of the NRS. Below, we describe these two primary components of the project: 
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1. Quality Assurance Study. To observe and collect information on various 
facets of the second year of NRS implementation, MPR and J&A staff made 
site visits to a nationally representative sample of 34 Head Start programs in fall 
2004 and to another nationally representative sample of 35 Head Start 
programs in spring 2005.^ We also visited a sample of 5 Migrant/Seasonal 
Head Start programs in spring 2005 and another sample of 5 Migrant/Seasonal 
programs in fall 2005. During these visits, staff observed the assessments of a 
random sample of children and interviewed program staff about their 
experiences implementing the NRS. In addition, in fall 2004 and spring 2005 
MPR visited subsamples of 5 programs to observe their local NRS assessor 
training sessions. 

2. System Development Support. MPR and J&A have supported the Head Start 
Bureau in assessing all aspects of the initial implementation of the NRS; 
augmenting the scope and enhancing the methods of NRS child assessment, 
data management, and training; and developing technically sound and useful 
ways to use NRS information in conjunction with other Head Start Bureau 
systems and resources to improve the quaKty and outcomes of Head Start 
services for all children. Information used to formulate recommendations for 
improvements has come from the NRS QuaKty Assurance Study, program 
practitioners’ views, input from the TWG, Head Start Bureau staff, other 
support contractors, and other federal agencies that often partner with 
ACYF / ACF on issues related to children’s programs and research. 

The purpose of this report is to provide updated findings from the QuaKty Assurance 
Study — focusing primarily on NRS implementation in spring 2005 — and make 
recommendations for system improvement. 

Quality of Assessments 

During our site visits, we examined the quaKty of assessment administration by 
conducting structured observations of a random sample of assessments. During these 
observations, trained site visitors scored children’s responses along with the assessors and 
coded errors in administration using a form developed by MPR that was based on the ChKd 
Assessment Certification Form developed by Westat for certifying NRS assessors. 

Using the certification procedures developed by Westat as a guide, we used error codes 
and comparison of site visitor and assessor scores to compute a certification score for each 



2 MPR selected 35 programs to visit in faU 2004. However, multiple hurricanes prevented one program 
from hosting a site visit. 
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assessment.^ We observed 348 assessments in spring 2005 — 316 in English and 32 in 
Spanish. 

As in earKer rounds of the NRS Quality Assurance Study, in spring 2005 we found that 
the majority of observed assessments met or exceeded the standard of quaUty used in 
certifying assessors. The average certification score was 93, well above the cutoff of 85, and 
90 percent of observed assessments exceeded the certification standard. The reHabiKty of 
assessment scale scores remained high, and the quaKty of the Spanish-language assessments 
observed (32 in 1 1 sites) was also high, on average. 

Errors due to coaching, incorrect gestures, and scoring errors were lower in spring 2005 
than in fall 2004. The decHne may reflect both changes to the assessment in spring 2005 
(particularly those that simplified gesturing) and the greater experience of the assessors. In 
addition, most of the programs reported that children were more comfortable and knew 
more in the spring, and their higher level of cooperation probably was one reason assessors 
coached less. In Year One, programs also reported that the assessment went much more 
smoothly in the spring. 

Although quality was high overall, some areas of the assessment were difficult for staff 
Modifications to the assessment easel or additional guidance and training might be helpful in 
these areas. Most of these areas of difficulty also were problematic in Year One. They 
include: 



• Setting up in a quiet area free of distractions 

• Avoiding coaching in the Simon Says section 

• Scoring the counting item correctly in the Early Math section 

Performance in these difficult areas had improved somewhat since Year One and since fall 
2004. Scoring errors on the counting item, however, were essentially unchanged — ^it was 
scored incorrectly on roughly one in three assessments. 

In several sites, assessors admitted that they had developed their own approach to issues 
that were not clear, seemingly unaware of the help available via the NRS helpline. Assessors 
stiU made script or scoring errors for selected questions for a modest proportion of 
assessments (10 to 15 percent). Gesturing errors, although reduced, remained common for 



3 In consultation with the Head Start Bureau, Westat developed the procedures and standards for 
certifying assessors to administer the NRS child assessment. To determine whether a trainee meets the 
standard, a certified NRS trainer observes the trainee adrninistering the assessment. The trainer records the 
frequency of scoring errors and of four types of administration errors: (1) straying from the script; 

(2) coaching; (3) non-neutral encouragement; and (4) other errors (pronunciation, inserting articles, or incorrect 
gesturing). Chapter II provides a more detailed description of the certification process and scoring procedures. 
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some questions — ^in part because not everyone followed the changes in procedure — but 
most observers judged that these errors had little effect on children’s responses. 

About two-thirds of programs reported that most children reacted positively to the 
child assessment. Assessors noted that some children enjoyed the one-on-one time with the 
assessor and Kked showing off what they knew. Children’s behavior was much less of a 
concern in the spring than it had been in the fall, a finding similar to Year One. The major 
behavioral issue was that children became bored or restless during the PPVT and/or the 
Letter Naming task and needed a lot of redirection. Over 40 percent of the programs 
thought that the assessment was too long. 

Almost all the programs (30 out of 35, or 86 percent) administered the child assessment 
to English language learners. Spanish was the most common other language spoken, 
followed by Chinese and Arabic. Assessors reported that most children could pass the 
Engksh language screener in the spring, even if they had not passed in the fall. Assessors 
also noted that it was awkward to conduct the Spanish assessment first for those classified as 
Spanish-speakers, as many children were used to speaking only Engksh at Head Start. Such 
children found it confusing to be tested in Spanish, and some refused to speak Spanish. 
Children’s reactions were similar in spring 2004. Assessors were also concerned about the 
burden on Spanish-speaking children of completing both versions of the assessment. 

The sample of programs visited in spring 2005 contained programs with smaller 
numbers of Spanish speakers per program than in the previous two rounds (although similar 
in numbers to fall 2003) and may provide useful descriptive information on the experiences 
of these types of programs. In these programs, assessors were confused about some aspects 
of assessment administration: for example, some were stiU giving the Engksh assessment 
before the Spanish assessment, while others were not aware that Engksh responses were 
acceptable for the Letter Naming task, except for the specifically Spanish letters. In 
addition, observers reported that a few of the Spanish assessors did not know Spanish weU 
or knew Spanish to speak but not to read. Conducting Spanish assessments in programs 
with just a few Spanish speakers may be an area that merits some attention in technical 
assistance and training. 

Most programs (30 out of 35 in the sample) assessed children with disabikties, using a 
wide range of accommodations for the child assessments. Some children who had 
experienced difficulty in the faU were able to complete the assessment in the spring, so only a 
very few children could not be assessed. However, in a few instances, assessors assessed a 
child who could have reasonably been exempted. Most of the programs that administered 
assessments to children with disabikties felt comfortable with the process and with the 
accommodations that they made for them. One-sixth of these programs, fewer than in the 
fall, desired more information on when to administer the NRS to children with disabikties 
and on appropriate accommodations for these testing situations. 
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Local Program Approaches to Training Assessors 

To ensure consistent administration of the NRS assessment. Head Start programs need 
to follow standardized procedures when training local staff Since the NRS was first 
implemented, the Head Start Bureau has provided local programs with standardized training 
materials and guidelines for conducting training. 

In spring 2005, all hut one of the sample programs provided refresher training, hut most 
did not adhere to the training guidelines. Only 14 percent of the sample programs included 
all of the required training elements — ^viewing and discussing the training video, distrihuting 
the Assessor’s Guide, and role-playing the assessment — ^in refresher training sessions. As a 
result, most training sessions were shorter than the recommended three to four hours. Two- 
thirds of the programs that needed to provide refresher training for Spanish assessors 
reported doing so, hut again most programs did not include all required components. 
Possible reasons why most programs did not adhere to the training protocol include 
concerns about staff burden, lack of time to complete assessments by the end of the 
program year, and trainers’ perceptions that assessors did not need refresher training. 

One-fifth of the sample programs trained new assessors in spring 2005. All of them 
provided separate training for new assessors and certified them during a practice assessment, 
but none of them followed the required training protocol for new assessors. Instead of 
providing a fuU-day training, most conducted the training in two to three hours. 

As in past rounds of site visits, the responses of local Head Start staff to the NRS 
materials and information provided by the Head Start Bureau were mixed. At least one staff 
member at most programs watched one of the NRS webcasts, but programs’ views of their 
usefulness was uneven. Lead NRS trainers tended to praise the spring 2005 training video; 
other staff thought it was too long and repetitive. Problems persisted with timely dehvery of 
training and assessment materials, and few programs took advantage of technical assistance 
offered through the help kne. Lack of thorough refresher training in most programs, 
coupled with a tendency not to seek technical assistance when questions arose, may account 
for some of the errors in administration and scoring described above. 



Local Approaches to Implementing the NRS 

Overall, the 35 Head Start programs we visited in spring 2005 took an approach to NRS 
implementation similar to that of programs visited in previous rounds of site visits. Program 
directors assigned a lead NRS trainer the responsibikty of overseeing implementation, 
including assigning, training, and certifying assessors; scheduKng and tracking the 
completion of assessments; overseeing quality assurance activities; and submitting score 
sheets by the deadline set by the Head Start Bureau. In more than a third of programs, the 
lead trainer was also responsible for maintaining the CBRS. 

Most programs maintained the same basic staffing structure for the NRS in the spring 
that they had instituted in the fall. When programs did make changes, most were due to 
staff turnover within the program rather than a rethinking of the program’s approach to the 
NRS. On average, the programs trained 14 assessors. Unlike the sample of programs we 
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visited in the faU, however, only a few programs in the spring rehed on teaching staff 
exclusively to conduct the assessments. Instead, programs assigned non-teaching staff or a 
mix of hoth teachers and other staff Although program staff saw many advantages to 
having teachers administer the assessment, many had concerns about teacher burden, 
reduction in instructional time, and the cost of hiring substitutes to cover for teachers when 
they were assessing children. 

Nearly all programs communicated with parents and Pohcy Councils about the NRS. 
While the means of communication varied, programs made an effort to inform parents of 
the assessment at the start of the program year. Two-thirds of programs obtained parents’ 
written consent for the NRS assessment, nearly twice as many as in previous rounds of site 
visits. Most programs had few or no parent refusals. Two programs, however, reported a 
large number of refusals; staff in both programs attributed these high refusal rates to 
intervention by members of their Policy Councils. Parent and PoKcy Council concerns were 
similar to those expressed in previous rounds of visits. They included parent requests to see 
their child’s individual results, concerns about bias of specific items, and questions about the 
purpose of the NRS, how the results would be used, and how the results would affect the 
Head Start program. 

Few Head Start directors estimated the monetary and in-kind costs of NRS 
implementation. However, a number of directors identified significant costs, such as staff 
time and travel costs, and costs for paying substitute teachers, contracted staff, and overtime 
for staff working on the NRS. 



Using the Computer-Based Reporting System 

As in past rounds of site visits, programs reported that the CBRS was easy to learn and 
use. Programs had adequate numbers of computers and Internet connections for accessing 
the system. Many programs used and hked the new features of the CBRS, especially the data 
copy and data import features. Most programs reported accessing the CBRS hotkne at least 
once and receiving prompt and helpful support from the hotkne staff Data managers made 
some suggestions for further enhancements to the CBRS, including reducing the cost of the 
data import feature, providing more support for using the data copy feature, keeping the 
CBRS open for data entry for longer periods of time, and adding other user-friendly features. 



Using the NRS for Local Program Improvement Efforts 

Across all Head Start staff who participated in the site visits, the majority found the 
2003—2004 Growth Report (that provides program-level information on children’s progress 
during the year) easy to understand, although some said it was confusing for non-technical 
audiences. Staff made several recommendations to improve the report’s content (for 
example, provide information on how subgroups of children performed) and format. In 
more than half of the sample programs, staff said that classroom-level reporting would be 
more useful in targeting classroom practices and resources; only two programs preferred 
reports only at the program level. In about half of the sample programs, staff said that their 
local child assessment results were more useful for informing program improvement 
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efforts — because they provided information on a broader range of child outcomes; they 
provided more timely information; and they provided information at the individual child, 
classroom, and center levels. 

More programs than in previous rounds of site visits reported using the NRS results, 
perhaps because by this time they had received the Growth Reports and had time to 
determine how they might be used. A majority of the sample programs shared NRS results 
with staff and key stakeholders such as managers, specialists, and teachers; Pohcy Councils; 
boards of directors; and, to a lesser degree, all parents. In addition, most programs have 
used the NRS results to identify areas in which children did not perform well and have taken 
that information into account when making decisions about program planning, curriculum, 
and teacher training. Some programs shared details on how they have used NRS data to 
modify classroom practices, such as spending more time on alphabet knowledge, hteracy 
development, and, to a lesser degree, counting and other early math skills. Because the NRS 
reports did not provide center-, classroom- or child-level information, 14 percent of the 
programs reported tracking the item responses of individual children using the Scantron 
answer sheets to make decisions on what areas needed improvement to better prepare 
children for kindergarten. 



Perspectives of Local Head Start Staff on the National Reporting System 

Local program staff raised several concerns about the NRS and its implications for 
future directions Head Start may take; most of these concerns also had been raised in 
previous rounds of site visits. The concern expressed most often by local staff was about 
how the NRS results would be used at the national level. Many staff said that they still did 
not have a clear understanding of the purpose of the NRS and its implications for local 
programs whose children did not perform well on the assessment. Local staff also expressed 
concern about whether the NRS results would accurately reflect program performance, the 
amount of staff time and financial resources dedicated to the NRS, and whether the 
assessment process might adversely affect some Head Start programs and children. A new 
concern raised in this round of site visits was how the NRS growth reports should be 
interpreted and how the information should be used by local programs. 

To improve NRS implementation, programs suggested that the Head Start Bureau share 
more information about how the results would be used, send training materials and outcome 
reports to programs sooner, provide more information about the assessment’s development 
and validity, and consult more with the Head Start community about future changes to the 
assessment. Some programs also requested written materials about the NRS for parents. 
Staff also suggested modifications to refresher training procedures, requested more support 
for lead NRS trainers, and asked for more training on a range of topics. 

Regarding the assessment battery, staff were spHt on whether new domains should be 
added. Many staff thought that at least one new domain — ^particularly social-emotional 
development — should be added, but at the same time they did not want the assessment to 
become longer. Program staff continued to recommend that the NRS be combined with the 
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local assessments. They also suggested changes to improve the Spanish-language version and 
to modify specific assessment procedures and items. 

Implications for System Improvement 

If the NRS is to he successful, local Head Start programs must he ahle to implement it 
well, accept it as a valid and reliable assessment, and understand how it can lead to program 
improvement. On the basis of findings from site visits to nationally representative samples 
of programs over the first two years of NRS implementation, and our understanding of 
concerns raised by the TWO, the Head Start Bureau, and others in the field, we suggest ideas 
for improving the NRS. These suggestions are of several types: (1) ideas for increasing 
communication with local Head Start programs to improve knowledge and understanding of 
the purposes of the system, (2) suggestions for improving access to and usefulness of NRS 
results for local programs, (3) providing support for administering the assessment (including 
guidance on assessing children in Spanish and children with options for supporting 
programs in administering the disabilities), and (4) ideas for changes to the assessment 
battery. 

Communication About Use of the Assessment 

Increased communication about the Head Start Bureau’s plans for the NRS would help 
address uncertainty about its purposes and how it will be used at the national level. 
Suggestions for improving communication follow: 



• Provide more information to Head Start Programs about how the Head Start 
Bureau will use the NRS results. 

• Provide a technical report or background materials about how the NRS 
assessment battery was developed, the vakdity and rehabikty of each task, and 
the appropriateness of comparing results from fall to spring and across years for 
individual children and programs, to be made available to the public. 

• Explore options for increasing consultation with Head Start practitioners on 
future changes to the NRS. 

• Provide written materials about the NRS for parents 

Access to and Usefulness of Assessment Results for Local Programs 

Program staff suggested several ways to make NRS outcome reports more useful to 
them and requested help in interpreting the results and implementing appropriate changes to 
their programs: 



• Send reports sooner after each round of data collection, preferably in time to 
use them in planning for the next program year. 
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• Consider providing results at the center level. 

• Provide programs with more guidance on how to use the NRS results for 
program improvement. 

• Provide access to resources to support local program improvement efforts 
undertaken in response to assessment results. 



Support for Administering the Assessment 

More timely shipment of training and assessment materials and modest changes in the 
NRS training protocols have the potential to help staff feel more confident and conduct the 
assessments more smoothly and accurately. Recommended changes include the following: 



• Ensure that programs receive materials and scheduKng information on time. 

• Provide a briefer format for refresher training focused on changes. 

• Take steps to ensure that all NRS assessors receive an updated Assessor’s Guide 
during training. 

• Consider requiring recertification of assessors during refresher training. 

• Provide more guidance on how to interact with children during the assessment. 

• Offer training support to programs that experience turnover in NRS trainers. 

Guidance for Assessing Children in Spanish 

Assessors and trainers continued to have some concerns about administering the 
Spanish version of the NRS and expectations for growth in Spanish-language skills. Some 
possible approaches to addressing these concerns include the following: 



• Allow more regional differences in Spanish terms used in the assessment. 

• Clarify expectations for growth in Spanish skills. 

Guidance on Assessing Children with Disabilities 

Most programs in the sample reported assessing at least one child with a disabikty 
during the first two years of NRS implementation. Although the majority of programs felt 
comfortable with the accommodations they made, some requested more guidance on how to 
assess children with disabilities. More guidance would be helpful in the following areas: 



• Clarify when a child should not be assessed because of a disability. 
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• Provide more information on when an assessment can be stopped. 

• Provide more examples of appropriate accommodations. 

Changes to the Assessment Battery 

Although staff were pleased with the changes in the NRS assessment in spring 2004, 
some of the changes made for fall 2004 and spring 2005 raised some concerns. In addition, 
some parts of the assessment that were a concern in Year One continued to be areas of 
concern to staff or to be associated with relatively high rates of error. In Kght of these 
concerns, it might be appropriate to review the assessment in the following areas: 



• Consider adding a measure of children’s social-emotional development. 

• Consider the use of manipulatives for Early Math Skills items that involve 
counting, addition, and subtraction. 

• Consider ekminating the item about money (El 4) and the pie question (El 6) 
from the Early Math Skills section. 

• Consider alternative approaches to the Letter Naming task. 
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